feat!: Reorganize pipelines and infra files into their respective folders #292

adlersantos · 2022-02-10T20:27:41Z

Description

This PR sets the stage for adding more categories (subfolders) in every datasets/$DATASET folder, such as the upcoming docs folder for the datasets' documentation set.

To do this, we need to change how the files and folders under every dataset is organized.

From

datasets/$DATASET/_terraform/
datasets/$DATASET/_images/
datasets/$DATASET/$PIPELINE_1/
datasets/$DATASET/$PIPELINE_2/

to introducing two levels infra and pipelines:

datasets/$DATASET/infra/
datasets/$DATASET/pipelines/_images/
datasets/$DATASET/pipelines/$PIPELINE_1/
datasets/$DATASET/pipelines/$PIPELINE_2/

which allows us to also add a docs folder. When the docset feature is ready, the hierarchy will look like

datasets/$DATASET/infra/
datasets/$DATASET/pipelines/
datasets/$DATASET/docs/

and we can keep adding other domain-specific folders as necessary without affecting infra- or pipelines-related stuff.

Checklist

Note: If an item applies to you, all of its sub-items must be fulfilled

leahecole

Non blocking nits. Otherwise, lgtm

leahecole · 2022-02-10T21:13:18Z

README.md

- GCS bucket to store final, downstream, customer-facing data
- Sometimes, for very large datasets, you might need to provision a [Dataflow](https://cloud.google.com/dataflow/docs) job
+- GCS bucket to store downstream data, such as those linked to in the [Datasets Marketplace](https://console.cloud.google.com/marketplace/browse?filter=solution-type:dataset).
+- Sometimes, for very large datasets that requires processing to be parallelized, you might need to provision a [Dataflow](https://cloud.google.com/dataflow/docs) (i.e. Apache Beam) job


nit

Suggested change

- Sometimes, for very large datasets that requires processing to be parallelized, you might need to provision a [Dataflow](https://cloud.google.com/dataflow/docs) (i.e. Apache Beam) job

- Sometimes, for very large datasets that requires processing to be parallelized, you might need to provision a [Dataflow](https://cloud.google.com/dataflow/docs) (Apache Beam) job

leahecole · 2022-02-10T21:14:46Z

README.md

@@ -16,12 +16,12 @@ Cloud-native, data pipeline architecture for onboarding public datasets to [Data

 # Environment Setup

-We use Pipenv to make environment setup more deterministic and uniform across different machines.
+We use Pipenv to make environment setup more deterministic and uniform across different machines. If you haven't done so, install Pipenv using the instructions found [here](https://pipenv-fork.readthedocs.io/en/latest/install.html#installing-pipenv).


Suggested change

We use Pipenv to make environment setup more deterministic and uniform across different machines. If you haven't done so, install Pipenv using the instructions found [here](https://pipenv-fork.readthedocs.io/en/latest/install.html#installing-pipenv).

We use Pipenv to make environment setup more deterministic and uniform across different machines. If you haven't done so, install Pipenv using these [instructions](https://pipenv-fork.readthedocs.io/en/latest/install.html#installing-pipenv).

nit - more screenreader friendly hyperlink

leahecole · 2022-02-10T21:16:41Z

README.md


-Consider this "dot" directory as your own dedicated space for prototyping. The files and variables created in that directory will use an isolated environment. All such directories are gitignored.
+Consider a dot directory as your own sandbox, specific to your machine, that's mainly used for prototyping. As will be seen later, this directory is where you will set the variables specific to your environment: such as actual GCS bucket names, GCR repository URLs, and secrets (we recommend using [Secret Manager](https://cloud.google.com/composer/docs/secret-manager) for this). The files and variables created or copied in the dot directories are isolated from the main repo, i.e. all dot directories are gitignored.


small nits

Suggested change

Consider a dot directory as your own sandbox, specific to your machine, that's mainly used for prototyping. As will be seen later, this directory is where you will set the variables specific to your environment: such as actual GCS bucket names, GCR repository URLs, and secrets (we recommend using [Secret Manager](https://cloud.google.com/composer/docs/secret-manager) for this). The files and variables created or copied in the dot directories are isolated from the main repo, i.e. all dot directories are gitignored.

We strongly recommend using a dot directory as your own sandbox, specific to your machine, that's mainly used for prototyping. This directory is where you will set the variables specific to your environment: such as actual GCS bucket names, GCR repository URLs, and secrets (we recommend using [Secret Manager](https://cloud.google.com/composer/docs/secret-manager) for this). The files and variables created or copied in the dot directories are isolated from the main repo, meaning that all dot directories are gitignored.

adlersantos added 7 commits February 10, 2022 15:01

feat: Revise generate_terraform and tests

421b302

feat: Revise generate_dag and tests

3db340c

feat: Revise deploy_dag and tests

4f84df1

feat: Revise tests for DAG conventions and integrity

e6d1352

add machine types guide to GKE cluster creation

0be16ba

revised README for new folder organization

91d48a3

Moved files into pipelines and infra folders

97825ab

adlersantos added revision: readme Improvements or additions to the README feature request New feature or request cleanup Cleanup or refactor code labels Feb 10, 2022

adlersantos requested review from happyhuman and leahecole February 10, 2022 20:27

adlersantos changed the title ~~!feat: Reorganize pipelines and infra files into their respective folders~~ feat!: Reorganize pipelines and infra files into their respective folders Feb 10, 2022

leahecole mentioned this pull request Feb 10, 2022

feat: Onboard Imaging Data Commons (IDC) v7 dataset #287

Merged

4 tasks

leahecole approved these changes Feb 10, 2022

View reviewed changes

happyhuman approved these changes Feb 10, 2022

View reviewed changes

happyhuman merged commit 7408d44 into main Feb 10, 2022

happyhuman deleted the new-dataset-subfolders branch February 10, 2022 22:15

release-please bot mentioned this pull request Feb 10, 2022

chore(main): release 3.0.0 #282

Merged

adlersantos mentioned this pull request Feb 11, 2022

docs: Revised the README #295

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Reorganize pipelines and infra files into their respective folders #292

feat!: Reorganize pipelines and infra files into their respective folders #292

adlersantos commented Feb 10, 2022 •

edited

Loading

leahecole left a comment

leahecole Feb 10, 2022

leahecole Feb 10, 2022

leahecole Feb 10, 2022

	- Sometimes, for very large datasets that requires processing to be parallelized, you might need to provision a [Dataflow](https://cloud.google.com/dataflow/docs) (i.e. Apache Beam) job
	- Sometimes, for very large datasets that requires processing to be parallelized, you might need to provision a [Dataflow](https://cloud.google.com/dataflow/docs) (Apache Beam) job

	We use Pipenv to make environment setup more deterministic and uniform across different machines. If you haven't done so, install Pipenv using the instructions found [here](https://pipenv-fork.readthedocs.io/en/latest/install.html#installing-pipenv).
	We use Pipenv to make environment setup more deterministic and uniform across different machines. If you haven't done so, install Pipenv using these [instructions](https://pipenv-fork.readthedocs.io/en/latest/install.html#installing-pipenv).


		Consider this "dot" directory as your own dedicated space for prototyping. The files and variables created in that directory will use an isolated environment. All such directories are gitignored.
		Consider a dot directory as your own sandbox, specific to your machine, that's mainly used for prototyping. As will be seen later, this directory is where you will set the variables specific to your environment: such as actual GCS bucket names, GCR repository URLs, and secrets (we recommend using [Secret Manager](https://cloud.google.com/composer/docs/secret-manager) for this). The files and variables created or copied in the dot directories are isolated from the main repo, i.e. all dot directories are gitignored.

feat!: Reorganize pipelines and infra files into their respective folders #292

feat!: Reorganize pipelines and infra files into their respective folders #292

Conversation

adlersantos commented Feb 10, 2022 • edited Loading

Description

Checklist

leahecole left a comment

Choose a reason for hiding this comment

leahecole Feb 10, 2022

Choose a reason for hiding this comment

leahecole Feb 10, 2022

Choose a reason for hiding this comment

leahecole Feb 10, 2022

Choose a reason for hiding this comment

adlersantos commented Feb 10, 2022 •

edited

Loading